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O. ^1 Noisy-channel coding 


@ Source coding (Data compression) 
92 Information content, entropy 
93 Typicality and the source coding theorem 
94 Symbol codes 
5 Symbol codes and Arithmetic coding 
@Noisy-channel coding 
26 Inference and 
Information measures for noisy channels 
#7 Capacity of a noisy channel 
#8 The noisy-channel coding theorem 


Definition of Capacity 


The Capacity of a channel 


is the maximum, over all input distributions P(x), 
of the mutual information: 


С = maxI(X;Y) 
PX 


The distribution PX that achieves the maximum is 
called the optimal input distribution. 


@Shannon's noisy channel coding theorem: 


Reliable (virtually error-free) communication is possible 
at rates up to C 


Noisy typewriter 


A A 
B B 
с< C 
D D 
E E 
Е Е Р(у=Е|т=б) = 1/3; 
G G P(y=G|2=G) = 1/3; 
H B P(y=H|2=G) = 1/3; 


‘What is the optimal input distribution? 


What is the capacity? 
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Non-confusable subset of inputs 


reliable communication is possible at C = log 9 bits 
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Shannon's noisy channel coding theorem 
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For any channel: 
Reliable (virtually error-free) communication is possible 
at rates up to C 
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(7,4) Hamming Code 
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H = |01 1 1 0 1 0| M=3 
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Valid transmissions t satisfy 


0 
Ht= | 0 | mod2 
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Valid transmissions t satisfy 
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(7,4) Hamming Code 
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Valid transmissions t satisfy 


0 
0 | mod2 


Received signal г = t +n 
Syndrome z = Hr = Hn. 
Syndrome decoder 2 — їй. 
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How we won the bent coin lottery Erobablitict T= 


To have a 99.99% chance of winning, 
we bought all the typical tickets 


How we won the bent coin lottery Erobablitict T= 


To have a 99.99% chance of winning, 
we bought all the typical tickets 
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How to prove good codes exist 


@Constructive proof 
Given required R < С, and e > 0 
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@Non-constructive proof 


Shannon's way of proving malnutrition 
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If average weight 

of all babies is < e, 

there must be (at least!) 
one baby with weight < e. 


Capacity 


The Capacity of a channel 
is the maximum, over all input distributions P(x), 
of the mutual information: 
С = maxI(X;Y) 
Px 


The distribution PY that achieves the maximum is 
called the optimal input distribution. 


@Shannon's noisy channel coding theorem: 
Reliable (virtually error-free) communication is possible at 
rates up to C 
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Proof for the Binary Symmetric Channel 
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@uLinear block code 

(Syndrome decoder 

(Quse typical noise vectors 
@Compute average probability of error 


How we won the bent coin lottery Probability of = f 


To have a 99.99% chance of winning, we bought all the typical tickets 
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If x is a fixed non-zero binary vector of length N 
and h is a random (1, 1) binary vector of length N, 
what is the probability that 


h'x = 0 mod 2? 


If x is a fixed non-zero binary vector of length N 


and H is a random (2, 2) binary matrix, 
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what is the probability that 
Hx=0 mod 2? 
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Homework recommendations 


(Noisy channels - Chapters 8, 9, 10 (10.1-10.4 only) 
^ Exercises 9.17 (p155); 10.12 (172); 15.12 (235) 
© and (if you want more practice) 15.11, 15.13, 15.15 


Olnvent a channel to pose to your colleagues: 
“'what's the capacity of this ?' 


